<!DOCTYPE html>
<html>
<head>
<title>周志华《机器学习》课后习题解答系列（六）：Ch5.7 - RBF网络实验</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
/* GitHub stylesheet for MarkdownPad (http://markdownpad.com) */
/* Author: Nicolas Hery - http://nicolashery.com */
/* Version: b13fe65ca28d2e568c6ed5d7f06581183df8f2ff */
/* Source: https://github.com/nicolahery/markdownpad-github */

/* RESET
=============================================================================*/

html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
  margin: 0;
  padding: 0;
  border: 0;
}

/* BODY
=============================================================================*/

body {
  font-family: Helvetica, arial, freesans, clean, sans-serif;
  font-size: 14px;
  line-height: 1.6;
  color: #333;
  background-color: #fff;
  padding: 20px;
  max-width: 960px;
  margin: 0 auto;
}

body>*:first-child {
  margin-top: 0 !important;
}

body>*:last-child {
  margin-bottom: 0 !important;
}

/* BLOCKS
=============================================================================*/

p, blockquote, ul, ol, dl, table, pre {
  margin: 15px 0;
}

/* HEADERS
=============================================================================*/

h1, h2, h3, h4, h5, h6 {
  margin: 20px 0 10px;
  padding: 0;
  font-weight: bold;
  -webkit-font-smoothing: antialiased;
}

h1 tt, h1 code, h2 tt, h2 code, h3 tt, h3 code, h4 tt, h4 code, h5 tt, h5 code, h6 tt, h6 code {
  font-size: inherit;
}

h1 {
  font-size: 28px;
  color: #000;
}

h2 {
  font-size: 24px;
  border-bottom: 1px solid #ccc;
  color: #000;
}

h3 {
  font-size: 18px;
}

h4 {
  font-size: 16px;
}

h5 {
  font-size: 14px;
}

h6 {
  color: #777;
  font-size: 14px;
}

body>h2:first-child, body>h1:first-child, body>h1:first-child+h2, body>h3:first-child, body>h4:first-child, body>h5:first-child, body>h6:first-child {
  margin-top: 0;
  padding-top: 0;
}

a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
  margin-top: 0;
  padding-top: 0;
}

h1+p, h2+p, h3+p, h4+p, h5+p, h6+p {
  margin-top: 10px;
}

/* LINKS
=============================================================================*/

a {
  color: #4183C4;
  text-decoration: none;
}

a:hover {
  text-decoration: underline;
}

/* LISTS
=============================================================================*/

ul, ol {
  padding-left: 30px;
}

ul li > :first-child, 
ol li > :first-child, 
ul li ul:first-of-type, 
ol li ol:first-of-type, 
ul li ol:first-of-type, 
ol li ul:first-of-type {
  margin-top: 0px;
}

ul ul, ul ol, ol ol, ol ul {
  margin-bottom: 0;
}

dl {
  padding: 0;
}

dl dt {
  font-size: 14px;
  font-weight: bold;
  font-style: italic;
  padding: 0;
  margin: 15px 0 5px;
}

dl dt:first-child {
  padding: 0;
}

dl dt>:first-child {
  margin-top: 0px;
}

dl dt>:last-child {
  margin-bottom: 0px;
}

dl dd {
  margin: 0 0 15px;
  padding: 0 15px;
}

dl dd>:first-child {
  margin-top: 0px;
}

dl dd>:last-child {
  margin-bottom: 0px;
}

/* CODE
=============================================================================*/

pre, code, tt {
  font-size: 12px;
  font-family: Consolas, "Liberation Mono", Courier, monospace;
}

code, tt {
  margin: 0 0px;
  padding: 0px 0px;
  white-space: nowrap;
  border: 1px solid #eaeaea;
  background-color: #f8f8f8;
  border-radius: 3px;
}

pre>code {
  margin: 0;
  padding: 0;
  white-space: pre;
  border: none;
  background: transparent;
}

pre {
  background-color: #f8f8f8;
  border: 1px solid #ccc;
  font-size: 13px;
  line-height: 19px;
  overflow: auto;
  padding: 6px 10px;
  border-radius: 3px;
}

pre code, pre tt {
  background-color: transparent;
  border: none;
}

kbd {
    -moz-border-bottom-colors: none;
    -moz-border-left-colors: none;
    -moz-border-right-colors: none;
    -moz-border-top-colors: none;
    background-color: #DDDDDD;
    background-image: linear-gradient(#F1F1F1, #DDDDDD);
    background-repeat: repeat-x;
    border-color: #DDDDDD #CCCCCC #CCCCCC #DDDDDD;
    border-image: none;
    border-radius: 2px 2px 2px 2px;
    border-style: solid;
    border-width: 1px;
    font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
    line-height: 10px;
    padding: 1px 4px;
}

/* QUOTES
=============================================================================*/

blockquote {
  border-left: 4px solid #DDD;
  padding: 0 15px;
  color: #777;
}

blockquote>:first-child {
  margin-top: 0px;
}

blockquote>:last-child {
  margin-bottom: 0px;
}

/* HORIZONTAL RULES
=============================================================================*/

hr {
  clear: both;
  margin: 15px 0;
  height: 0px;
  overflow: hidden;
  border: none;
  background: transparent;
  border-bottom: 4px solid #ddd;
  padding: 0;
}

/* TABLES
=============================================================================*/

table th {
  font-weight: bold;
}

table th, table td {
  border: 1px solid #ccc;
  padding: 6px 13px;
}

table tr {
  border-top: 1px solid #ccc;
  background-color: #fff;
}

table tr:nth-child(2n) {
  background-color: #f8f8f8;
}

/* IMAGES
=============================================================================*/

img {
  max-width: 100%
}
</style>
</head>
<body>
<p>相关答案和源代码托管在我的Github上：<a href="https://github.com/PY131/Machine-Learning_ZhouZhihua">PY131/Machine-Learning_ZhouZhihua</a>.</p>
<h2>RBF神经网络实验</h2>
<blockquote>
<p><img src="Ch5/5.7.png" /></p>
</blockquote>
<p>注：本题程序基于Python实现（<a href="https://github.com/PY131/Machine-Learning_ZhouZhihua/tree/master/ch5_neural_networks/5.7_RBF_BP">这里查看完整代码和数据集</a>）。</p>
<h3>1. RBF网络基础</h3>
<p>RBF网络采用RBF（Radial Basis Function函数）作为隐层神经元激活函数，是一种局部逼近神经网络，下面先分析其激活函数RBF，然后分析RBF神经网络的结构。</p>
<h4>1.1. 径向基函数(RBF)</h4>
<p>径向基函数是一类取值依赖样本于到中心点距离的函数，本题基于常用的<strong>高斯径向基函数</strong>（gaussian RBF）开展实验。下面是高斯径向基函数形式书p108式(5.19)：</p>
<p><img src="Ch5/5.7.1.png" /></p>
<p>这里的 β 为尺度系数， c_i 为中心点（维度由输入决定），函数的取值取决于样本 x 到中心点的距离（2-范数），该函数的参数为 (β, c_i)。</p>
<p>如下图示为高斯径向基函数示意图（<a href="https://github.com/PY131/Machine-Learning_ZhouZhihua/blob/master/ch5_neural_networks/5.7_RBF_BP/draw_RBF.py">绘图程序</a>）：</p>
<p><img src="Ch5/5.7.rbf_fig1.png" /></p>
<h4>1.2. RBF网络</h4>
<p>RBF神经网络一般指一种单隐层前馈神经网络，它使用径向基函数作为隐层神经元激活函数，而输出是隐层输出的线性组合，网络结构示意如下：</p>
<p><img src="Ch5/5.7.net_fig1.png" /></p>
<p>参考书p108式(5.18)，该神经网络的输出为：</p>
<p><img src="Ch5/5.7.2.png" /></p>
<p>进一步分析，一般函数均可表示成一组基函数的线性组合，而RBF网络相当于用隐层神经元构建了这样一组基函数，由输出层进行线性组合，从而实现函数逼近的功能。</p>
<h3>2. RBF网络实现</h3>
<p><a href="https://github.com/PY131/Machine-Learning_ZhouZhihua/blob/master/ch5_neural_networks/5.7_RBF_BP/RBF_BP.py">查看完整代码</a></p>
<p>这里RBF神经网络建模的过程分一下两步：</p>
<pre><code>1. 确定神经元对应的高斯径向基函数中心 c ；
2. 利用BP算法来训练剩余参数 w, β ；
</code></pre>

<p>下面依次讨论其实现：</p>
<h4>2.1. RBF中心获取</h4>
<p>RBF的中心参数 c 的获取方法有以下一些：</p>
<ul>
<li><strong>从输入数据样本中抽取</strong>，这就要求输入样本具有较好的代表性；</li>
<li><strong>自组织生成</strong>，如<strong>聚类</strong>法生成，采用聚类中心来作为中心参数 c ，同时根据各中心的距离来初始化尺度系数 β ；</li>
</ul>
<h4>2.2. RBF-BP算法推导</h4>
<p>参考<a href="http://blog.csdn.net/snoopy_yuan/article/details/70765238">神经网络基础 - Python编程实现标准BP算法</a>，这里隐层激活函数为RBF，输出层神经元数为 1 ，激活函数为 y=f(x)=x 。</p>
<p>BP算法采用梯度下降法进行参数迭代更新，参考书p102-103，进行RBF-BP算法中基于梯度下降的参数更新推导如下：</p>
<p><img src="Ch5/5.7.rbf_bp_1.png" /></p>
<p>在完成基础推导之后，给出<strong>RBF网络的BP算法</strong>如下所示：</p>
<p><img src="Ch5/5.7.rbf_bp_2.png" /></p>
<h4>2.3. RBF-BP算法实现</h4>
<p>样例代码如下：</p>
<pre><code>```python
def BackPropagateRBF(self, x, y):
    '''
    the implementation of special BP algorithm on one slide of sample for RBF network
    @param x, y: array and float, input and output of the data sample
    '''

    # dependent packages
    import numpy as np 

    # get current network output
    self.y = self.Pred(x)

    # calculate the gradient for hidden layer 
    g = np.zeros(self.h_n)
    for h in range(self.h_n):
        g[h] = (self.y - y) * self.b[h]    

    # updating the parameter
    for h in range(self.h_n):
        self.beta[h] += self.lr * g[h] * self.w[h] * np.linalg.norm(x-self.c[h],2)
        self.w[h] -= self.lr * g[h]
```
</code></pre>

<h3>3. 异或问题实验</h3>
<p><a href="https://github.com/PY131/Machine-Learning_ZhouZhihua/blob/master/ch5_neural_networks/5.7_RBF_BP/Test_XOR.py">查看完整代码</a></p>
<h4>3.1. 准备数据</h4>
<p>首先基于numpy.array生成异或数据，该数据为2输入，1输出，如下所示：</p>
<p>样例代码：</p>
<pre><code>```python
# train set
X_trn = np.random.randint(0,2,(100,2))
y_trn = np.logical_xor(X_trn[:,0],X_trn[:,1])
```
</code></pre>

<p>样例数据：</p>
<pre><code>&gt;&gt;&gt; X_trn
array([[0, 0],
       [1, 1],
       ...
&gt;&gt;&gt; y_trn
array([False, False, ...
</code></pre>

<h4>3.2. 参数之-RBF中心点</h4>
<p>这里由于采用异或数据，其中心点可以简单设置如下：</p>
<pre><code>centers = np.array([[0,0],[0,1],[1,0],[1,1]])
</code></pre>

<p>同时取隐节点数目为4。</p>
<h4>3.3. 生成模型并训练</h4>
<p>样例代码如下：</p>
<pre><code>```python
# construct the network
rbf_nn = RBP_network()  # initial a BP network class
rbf_nn.CreateNN(4, centers, learningrate=0.05)  # build the network structure

# parameter training（这里迭代10次）
for i in range(10): 
    rbf_nn.TrainRBF(X_trn, y_trn)
```
</code></pre>

<p>绘制出训练过程中的均方误差变化曲线如下图：</p>
<p><img src="Ch5/5.7.covergence_curve.png" /></p>
<p>由上图可以看到，曲线收敛十分迅速，说明这里的RBF网络训练异或数据集十分轻松，这也和我们所生成的数据的完备无误有关。</p>
<h4>3.4. 测试模型</h4>
<p>按照训练集数据生成方法生成测试集数据，通过模型预测，得出结果如下：</p>
<pre><code>test error rate: 0.000
</code></pre>

<p>即测试错误率为0，可知我们的模型预测十分准确的，泛化性能优秀（主要得益于XOR预测模型对RBF网络来说太过简单）。</p>
<h3>4. 小结</h3>
<p>回顾RBF网络工作原理，如参考书p108式(5.18)-(5.19)。RBF网络建模类似于非线性模型中的基函数建模，进一步，我们可将其与广义可加模型联系起来。RBF网络采用径向基函数作为单隐层激活函数（即核函数），又可将其与SVM with RBF kernel联系起来。</p>
<p>回顾RBF网络实现过程，我们或可将该方法视为一种半监督的学习方法，具体有：</p>
<ul>
<li>step 1：无监督的学习，从数据中获取中心参数，常用聚类方法；</li>
<li>step 2：有监督的学习，基于数据训练参数，过程一般基于BP算法实现；</li>
</ul>
<h3>5. 参考</h3>
<p>下面列出一些参考内容：</p>
<ul>
<li>基础知识：<a href="https://wenku.baidu.com/view/c61384dda58da0116c174946.html">绝对经典RBF神经网络 - 百度文库</a></li>
<li>进阶知识：<a href="https://www.zhihu.com/question/44328472">RBF神经网络和BP神经网络有什么区别 - 知乎</a></li>
<li>可视化：<a href="http://matplotlib.org/1.4.3/mpl_toolkits/mplot3d/index.html#toolkit-mplot3d-index">matplotlib绘制3D图 - matplotlib官网</a></li>
</ul>

</body>
</html>
<!-- This document was created with MarkdownPad, the Markdown editor for Windows (http://markdownpad.com) -->
